Staleness-Aware Async-SGD for Distributed Deep Learning

This paper investigates the effect of stale (delayed) gradient updates within the context of asynchronous stochastic gradient descent (Async-SGD) optimization for distributed training of deep neural networks. We demonstrate that our implementation of Async-SGD on a HPC cluster can achieve a tight bound on the gradient staleness while providing near-linear speedup. We propose a variant of the SGD algorithm in which the learning rate is modulated according to the gradient staleness and provide theoretical guarantees for convergence of this algorithm. Experimental verification is performed on commonly-used image classification benchmarks: CIFAR10 and ImageNet to demonstrate the effectiveness of the proposed approach. Additionally, our experiments show that there exists a fundamental tradeoff between model accuracy and runtime performance that places a limit on the maximum amount of parallelism that may be extracted from this workload under the constraints of preserving the model quality.